-
Notifications
You must be signed in to change notification settings - Fork 67
feat(executors): Add Java connector support with automatic JRE management #719
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
…ment - Add JavaExecutor class for managing Java connectors - Implement automatic JRE download and caching in ~/.airbyte/java - Support Zulu 21 JRE with dynamic OS/architecture detection - Integrate with existing executor factory pattern - Remove Java connector warning from registry - Add support for connector tar file execution - Tested with source-mssql connector --spec command Co-Authored-By: AJ Steers <[email protected]>
Co-Authored-By: AJ Steers <[email protected]>
Original prompt from AJ Steers:
|
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
- Update get_connector_executor to accept use_java_tar parameter - Support Path, str, bool, and None types for use_java_tar - Implement logic for bool True/False similar to docker_image=True - Address GitHub PR feedback from @aaronsteers Co-Authored-By: AJ Steers <[email protected]>
- Rename connector_tar_path to use_java_tar in JavaExecutor.__init__ - Update parameter documentation - Prepare for bool value handling logic Co-Authored-By: AJ Steers <[email protected]>
- Support Path, str, bool, and None types for use_java_tar - Add logic to handle True/False values for auto-location - Update type hints and documentation - Complete implementation of GitHub PR feedback Co-Authored-By: AJ Steers <[email protected]>
- Update JavaExecutor call to use use_java_tar parameter - Fix mypy type checking error - Complete implementation of GitHub PR feedback Co-Authored-By: AJ Steers <[email protected]>
- Remove requirement for use_java_tar when no install method specified - Java connectors now fall back to Docker like other connector types - Address GitHub PR feedback from @aaronsteers Co-Authored-By: AJ Steers <[email protected]>
airbyte/_executors/util.py
Outdated
pip_url = metadata.pypi_package_name | ||
pip_url = f"{pip_url}=={version}" if version else pip_url | ||
case InstallType.JAVA: | ||
docker_image = True |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's make this a bit more robust...
When default install type is Java:
- If
use_java_tar
is False: fallback to Docker. - If
use_java_tar
is None and Docker is available: use Docker. - Else (
use_java_tar
is Truthy): use Java.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I forgot to mention the case where use_java_tar
is None and Docker is available (2b). In that case, we can default to Docker (for now) as the more stable and proven option when Docker is available.
Document these cases for the user within the docstring of the public function(s)/method(s), and within the file docstring for the java
module.
… documentation - Add comprehensive fallback logic for Java connectors in get_connector_executor - When use_java_tar=False: explicitly use Docker - When use_java_tar=None and Docker available: prefer Docker for stability - When use_java_tar=None and Docker unavailable: fallback to Java with auto-detection - When use_java_tar is truthy: use Java executor - Add detailed docstring documentation explaining fallback behavior - Add module-level docstring to java.py explaining fallback logic - Import is_docker_installed at module level for clean code structure Addresses GitHub PR comment requesting more robust fallback logic and documentation. Co-Authored-By: AJ Steers <[email protected]>
Co-Authored-By: AJ Steers <[email protected]>
Closing due to inactivity for more than 7 days. Configure here. |
- Add JavaExecutor with automatic JRE management and TAR extraction - Add use_java parameter (None/True/False/Path) for Java execution control - Add use_java_tar parameter (None/Path) for connector TAR file location - Implement fallback logic: use_java_tar implies use_java=True when set - Add comprehensive documentation and error handling - Create source-snowflake example demonstrating Java connector usage - Copy implementation from PR #719 with updated dual-parameter API Requested by: @aaronsteers Co-Authored-By: AJ Steers <[email protected]>
feat(executors): Add Java connector support with automatic JRE management
Summary
This PR adds comprehensive Java connector support to PyAirbyte by implementing a new
JavaExecutor
class that automatically downloads and manages Zulu JRE installations. The implementation follows the working logic from the Airbyte shell script and integrates seamlessly with PyAirbyte's existing executor framework.Key Features:
~/.airbyte/java/{os}-{arch}/
directoriesuse_java_tar
parameter with Path, str, bool, and None typesAPI Usage:
Review & Testing Checklist for Human
This is a medium-risk change introducing new platform-specific functionality:
source-mssql
) using gradlew and test the complete flow from tar installation to--spec
executionuse_java_tar
parameter types work as expected, especially the bool=True case for auto-locationRecommended test plan:
cd ~/repos/airbyte && ./gradlew :airbyte-integrations:connectors:source-mssql:build
ab.get_source("source-mssql", use_java_tar="path/to/airbyte-app.tar").spec()
Diagram
Notes
use_java_tar=True
case currently sets path to None - full auto-location logic may need additional implementation based on registry metadata.Session Info: